fix(scanner): include adjudicator token usage by vineethsai7 · Pull Request #151 · cisco-ai-defense/skill-scanner

vineethsai7 · 2026-08-03T20:16:41Z

What changed

track token usage for every successful adjudicator LiteLLM completion
include adjudicator input, output, and total tokens in ScanResult.llm_usage
preserve billed usage when an adjudicator response cannot be parsed or later processing fails
document that aggregate usage includes analyzer, meta-analysis, and adjudication calls

Why

PR #146 aggregated the regular LLM and meta-analyzer token usage, but PR #139's adjudicator calls LiteLLM directly and its usage was not included. Scans with --adjudicate therefore under-reported total LLM consumption.

Validation

79 focused tests passed across adjudicator, CLI usage, request-handler, and model serialization suites
post-rebase adjudicator and CLI integration tests passed
Ruff and repository hooks passed
CodeRabbit CLI raised 0 issues

Summary by CodeRabbit

New Features
- Added token-usage tracking for adjudication calls.
- Scan-level LLM usage totals now include adjudication input, output, and total tokens, including calls that return malformed results or encounter errors.
Documentation
- Updated architecture and JSON output documentation to describe adjudication usage aggregation and reporting conditions.
Tests
- Added coverage for token accumulation and combined analyzer and adjudicator usage totals.

coderabbitai · 2026-08-03T20:16:49Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: fb3334b2-dd5d-432d-8cfa-2e39358c8160

📥 Commits

Reviewing files that changed from the base of the PR and between 6155f06 and cdb5614.

📒 Files selected for processing (1)

docs/reference/output-formats.md

🚧 Files skipped from review as they are similar to previous changes (1)

docs/reference/output-formats.md

📝 Walkthrough

Walkthrough

The Adjudicator now tracks LLM token usage for each run. SkillScanner adds this usage to aggregate scan totals, including failed adjudication calls. Documentation and tests cover the updated accounting.

Changes

Adjudicator LLM usage aggregation

Layer / File(s)	Summary
Adjudicator usage tracking and validation `skill_scanner/core/analyzers/adjudicator.py`, `tests/test_adjudicator.py`	The Adjudicator resets and accumulates token usage for each run. Usage is recorded before response parsing, including malformed responses. Tests cover accumulation and scanner aggregation.
Scan integration and output documentation `skill_scanner/core/scanner.py`, `docs/architecture/analyzers/adjudicator.md`, `docs/reference/output-formats.md`	`SkillScanner` records Adjudicator usage in a `finally` block and adds it to aggregate scan totals. Documentation describes adjudication usage in `llm_usage` output.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant SkillScanner
  participant Adjudicator
  participant LLMProvider
  participant ScanResult
  SkillScanner->>Adjudicator: run adjudication
  Adjudicator->>LLMProvider: request adjudication
  LLMProvider-->>Adjudicator: response with token usage
  Adjudicator-->>SkillScanner: return or raise
  SkillScanner->>SkillScanner: add usage in finally
  SkillScanner->>ScanResult: emit aggregate llm_usage

Possibly related issues

feat: expose LLM token usage in ScanResult JSON output #136 — Tracks and aggregates LLM token usage for adjudicator and analyzer calls.

Possibly related PRs

cisco-ai-defense/skill-scanner#146 — Adds related LLM token-usage aggregation that this PR extends to adjudicator calls.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely describes the main change: including adjudicator token usage in scanner totals.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch agent/include-adjudicator-token-usage

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

coderabbitai

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docs/reference/output-formats.md`:
- Line 114: Correct the llm_usage documentation to state that the field is
included only when non-zero provider usage is reported, rather than whenever an
LLM call is made; alternatively, update the scanner’s usage aggregation to track
executed calls and preserve the documented call-based presence condition.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: e2b51637-1a0b-4c64-a222-4afcc7b1ca24

📥 Commits

Reviewing files that changed from the base of the PR and between 5dca0c5 and 6155f06.

📒 Files selected for processing (5)

docs/architecture/analyzers/adjudicator.md
docs/reference/output-formats.md
skill_scanner/core/analyzers/adjudicator.py
skill_scanner/core/scanner.py
tests/test_adjudicator.py

fix(scanner): include adjudicator token usage

6155f06

vineethsai7 marked this pull request as ready for review August 3, 2026 20:24

coderabbitai Bot reviewed Aug 3, 2026

View reviewed changes

Comment thread docs/reference/output-formats.md Outdated

docs: clarify LLM usage output condition

cdb5614

vineethsai7 merged commit b51e3d1 into main Aug 3, 2026
12 checks passed

vineethsai7 deleted the agent/include-adjudicator-token-usage branch August 3, 2026 20:47

Navigation Menu

Uh oh!

fix(scanner): include adjudicator token usage - #151

fix(scanner): include adjudicator token usage#151
vineethsai7 merged 2 commits into
mainfrom
agent/include-adjudicator-token-usage

vineethsai7 commented Aug 3, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

coderabbitai Bot commented Aug 3, 2026 •

edited

Loading

Walkthrough

Changes

Sequence Diagram(s)

Uh oh!

coderabbitai Bot left a comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

vineethsai7 commented Aug 3, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changed

Why

Validation

Summary by CodeRabbit

Uh oh!

coderabbitai Bot commented Aug 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Sequence Diagram(s)

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

vineethsai7 commented Aug 3, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Aug 3, 2026 •

edited

Loading